This invention relates to methods and systems for evaluating creative works, information, and products (collectively “works”). The invention relates specifically to methods and systems of quality review.
Quality assessment has long played an important role in the development and success of businesses, individuals, entertainment, and products. Proper identification of a quality work, individual, business, or organization is pivotal to the success of many ventures. Industries such as consumer products and services, entertainment, news/media, health care, publishing, and corporate knowledge management, rely on quality assessment for decision making. Unfortunately, the quality assessment process has generally been time-intensive and/or costly. This is in part due to the consistently higher supply of works compared to the average demand for those works.
This supply of works is provided by Creators. Creators are those who develop potentially valuable works, products, or information. Seekers are those wanting quality products, works or information to produce, market, or use. With improving communication systems, especially the Internet, the ratio of available creators to seekers has increased dramatically. It is typical for a seeker of a particular work to be confronted with selecting a creator of a work from a pool of hundreds or thousands, or more.
Selecting a quality work from such large pools can be overwhelming and discouraging for the seeker. For a given pool of a particular type of work, there typically exists several high quality works. The time expenditure to assess the quality of each work, however, is costly. Many seekers have ceased reviewing unsolicited works from creators because of the sheer volume of such works. Word-of-mouth and personal networks still prevail as methods to find desired works. Personal networks can be effective, but the individuals within the network do not normally review all works from a given pool. Selection through a personal network can result in sub-par works being selected while many quality works remain unreviewed and undiscovered. Like fishing from a pond instead of an ocean, a seeker's probable catch will be smaller than what it could be. Thus the seeker fills the demand without realizing the potential of the supply, while the average creator has little hope of getting noticed. An efficient system of assessing the quality of an ocean of works is desirable.
Prior attempts at providing a system for assessing quality include using an expert review system. In an expert review system, experts specializing in evaluation are compensated for reviewing works and reporting on the quality of each work. While the quality of the review itself is high, expert review systems have disadvantages. One disadvantage is the high cost. Seekers need to compensate expert reviewers under such a system; and decreasing the number of expert reviewers only increases the time commitment. With the expert review process slow or costly, the time savings in using an expert reviewer in finding a quality work is overshadowed by the cost of using expert reviewers. The price of having all works from a pool reviewed can easily surpass any expected return on investment from the eventually discovered work. Some expert review systems are funded by creators, but not all creators have the resources to fund such systems. Finally, the expert review system suffers from a small sampled portion. Whether seekers or creators fund the system, only a portion of the pool is reviewed which means many quality works remain undiscovered.
Another attempt at providing an effective system for assessing quality is a computerized review system. In a computerized review system, evaluation by humans is replaced with machines, computer software or the like. Such computerization enables an entire pool of works to be evaluated at a low cost—but at the expense of accuracy. While some properties of a work can be accurately reviewed by a computer, other properties cannot. Many properties of a work need evaluation at an emotional level to determine human appeal. Software has not been developed to a level where it can review a work and accurately return a human's probable emotional response.
There also exists peer review systems to assess quality. Peer review systems harnesses the most plentiful resource available—the creators' time. The collective resources of the creators fuel the review process. Peer review systems operate by providing a forum in which creators review each others works. The benefit is that Seekers can quickly browse top-rated works, all of which have been evaluated by humans. Operating such a system through a computer network makes large-scale collaboration relatively inexpensive. While this low-cost system produces better results than automated evaluation, current peer review systems suffer from a number of disadvantages.
One disadvantage of peer review systems is the enormous potential for abuse. Since creators are reviewing the works of competing creators, there is an inherent conflict of interest. A practice common to peer review systems is “panning.” Panning is rating all the works of one's peers with low scores to improve the relative ranking of one's own works. Another common practice is creating multiple, fictitious user accounts to rate one's own work at the top of the scale. Some peer review systems allow solicitation of reviews, but this encourages users to merely swap good ratings. With a high susceptibility to abuse and fraud, the quality of results from such peer-review systems suffers.
Another disadvantage of peer review systems is providing an accurate and reliable ranking system using reviewers who are not expert reviewers. Traditionally, peer review systems have used scalar method of rating works. For example, a reviewer is asked to rate a work on a scale of 1-10. Averaging the individual ratings from reviewers provides a consensus, but this erroneously assumes that the evaluation skills of each peer reviewer are equal. Such an erroneous assumption often yields misleading or inaccurate results. The scalar method also suffers from dead-ends of the scale. If a reviewer scores an item as “10” on a scale of 1 to 10, and the next reviewed item is better than the last item scored as “10,” then entered scores must be changed to compensate for the inaccuracy.
Another measurement technique is a simple relative measurement scale. For example, a reviewer is asked to choose the better of A vs. B. Results are tallied from several A vs. B comparisons. While there are no dead ends with simple relative measurements, this technique less efficiently finds a consensus.
There are several attempts in developing a networked peer review system. Zoetrope Virtual Studios (zoetrope.com) was one of the first websites on the Internet to use peer review technology. Originally just a website where writers could submit their stories to the short magazine “American Zoetrope,” Zoetrope began to let writers read and rate each other's stories to help the editors in their talent search. Zoetrope expanded to include screenplays, poetry and other written works with Screenwriting its most popular category.
Zoetrope makes a modest attempt at solving the problem of inaccurate ratings. It gives each user a short list of works upon joining the site. The user must choose four works from this list before being allowed to upload their own work. The user's work, then, is only available for review for thirty days after upload. After that thirty-day period, the user must solicit their fellow members to review their work. This solicitation, however, allows users to swap good ratings—a problem rampant on Zoetrope. While a number of top-ranked works on Zoetrope have been optioned and/or otherwise sold, most users complain about its ineffective rating system.
Project Greenlight (projectgreenlight.com) holds annual contests to find a director and a screenplay for a funded film production. The film production process is recorded as a reality television show complete with dramatic nuggets of director/writer conflict. The first round of each contest requires each contestant to read and review a certain number of their peers' scripts. Unfortunately, Project Greenlight's peer review engine does not protect adequately against panning and other fraudulent reviews. Each reader is required to answer specific questions to make sure that each reader had indeed read the script, but that is the extent of the quality safeguards.
Trigger Street (triggerstreet.com) uses a peer review system to evaluate short films and screenplays. No mechanism is in place to ensure accurate ratings making its ranked lists of questionable quality. Additionally, this peer review system benefits only one film production company, each user must give Trigger Street an exclusive 90-day option on any work that makes the top lists, and no compensation is given to the writer for this option. Another problem with Trigger Street is that a work must receive ten reviews before it can even make the ranked list. Since users can pick the work they choose to rate, most of them choose from the top of the list. This makes it difficult for a user to equitably get his works on the list.
Therefore, what is needed is a collaborative evaluation system that minimizes the work load of the evaluators while producing high quality evaluations from a large pool of works. What is further needed is a collaborative evaluation system that accurately calculates ranked works, prevents fraudulent reviews, and provides an incentive for quality reviews.